Joint Crowdsourcing of Multiple Tasks

Introduction Allocating tasks to workers so as to get the greatest amount of high-quality output for as little resources as possible is an overarching theme in crowdsourcing research. Among the factors that complicate this problem is the lack of information about the available workers’ skill, along with unknown difficulty of the tasks to be solved. Moreover, if a crowdsourcing platform customer is limited to a fixed-size worker pool to complete a large batch of jobs such as identifying a particular object in a collection of images or comparing the quality of many pairs of artifacts in crowdsourcing workflows, she inevitably faces the tradeoff between getting a few of these tasks done well or getting many done poorly. In this paper, we propose a framework called JOCR (Joint Crowdsourcing, pronounced as “Joker”) for analyzing joint allocations of many tasks to a pool of workers. JOCR encompasses a broad class of common crowdsourcing scenarios, and we pose the challenge of developing efficient algorithms for it. In the settings modeled by JOCR, a customer needs to get answers to a collection of multiple-choice questions and has a limited worker pool at her disposal. Each question has a certain level of difficulty (possibly unknown) and each worker has a certain level of ability (possibly unknown as well). The chance of a worker answering a question incorrectly is an increasing function of question difficulty and decreasing function of worker skill, as described in (Dai, Mausam, and Weld 2010). Adopting the “consensus task” model of (Kamar, Hacker, and Horvitz 2012), we allow each question to be assigned to several workers, whose responses can be aggregated in order to increase the chance of correct answer. Last but not least, the framework admits constraints on the number of questions that can be allocated per person that prevent any given worker from getting overwhelmed. We describe several possible optimization problems formalized by JOCR and suggest promising methods for solving them approximately.